1^20Rec'dPCT/PTO 23 JUN 2006 

DESCRIPTION 

MOVING PICTURE ENCODING METHOD AND APPARATUS 
Technical Field: 

5 The present invention relates to a moving picture encoding technique, 

and in particular relates to a moving picture encoding method and apparatus 
for performing a multi-frame motion prediction. 
Background Art: 

Figure 1 is a block diagram showing a configuration of a conventional 

10 typical encoding apparatus for encoding a moving picture signal. 

The encoding apparatus shown in Fig. 1 includes a local decoding 
apparatus, and is provided with frequency conversion unit 101, quantization 
unit 102, variable length encoding unit 103, dequantization unit 104, inverse 
frequency conversion unit 105, frame memory 106, intra-frame prediction unit 

15 107, motion compensation unit 108, motion estimation unit 109, buffer 110, 
and code amount control unit 111. The encoding apparatus further includes 
subtractor 121, switch 122, and adder 123. 

An input picture frame is fed into the encoding apparatus and is divided 
into a plurality of blocks. A predicted value by an intra-frame prediction or an 

20 inter-frame prediction is subtracted from the divided block by subtractor 121. 
Here, the intra-frame prediction is a technique for predicting a current picture 
by using a reconstructed area of the current frame to be encoded, and the 
inter-frame prediction is a technique for predicting a current picture by using a 
picture frame that is previously reconstructed. The picture block in which the 

25 predicted value by the intra-frame prediction or the inter-frame prediction is 
subtracted is called a prediction error. Incidentally, a picture frame in which all 



blocks in a frame to be encoded are encoded by only the intra-franne 
prediction for producing predicted values from adjacent pixels in the same 
frame to be encoded, is called an l-picture. A picture frame coded by using 
both the intra-frame prediction and the inter-frame prediction is called a P- 
5 picture. Further, in the inter-frame prediction, a picture frame coded by 
referring to a plurality of picture frames, which are input before and after the 
current encoded frame, is called a B-picture. 

Generally, in the moving picture data to be encoded, l-pictures are set 
at constant intervals, and a section that is separated by the l-picture and 

10 includes a plurality of frames is called a GOP (Group Of Picture). Definitions 
of the I-, P-, and B-pictures and the GOP are stipulated by the MPEG (Motion 
Picture Expert Group), which is the moving picture encoding standard in the 
international standardization. 

Then, the prediction error is converted into a frequency domain by 

15 frequency conversion unit 101 . The prediction error converted into the 
frequency domain is quantized by quantization unit 102. The quantized 
prediction error, i.e., a conversion coefficient is entropy-encoded by variable 
length encoding unit 1 03 and is stored in buffer 1 1 0, Buffer 110 outputs the 
stored occurring code, i.e., a bitstream with predetermined timing. Also, the 

20 quantized prediction error is returned to the prediction error of the original 
spatial domain again as a local decoding process by dequantization unit 104 
and inverse frequency conversion unit 105. Further, the predicted value is 
added to the prediction error returned to the spatial domain by adder 123, and 
the prediction error is stored in frame memory 106 as a reconstructed picture. 

25 The reconstructed picture that has been stored in frame memory 106 is 

referred by intra-frame prediction unit 107. motion compensation unit 108 and 
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motion estimation unit 1 09 to produce the predicted value. Therefore, the 
reconstructed picture stored in frame memory is also called a reference frame. 

Intra-frame prediction unit 107 performs the intra-prediction in 
accordance with the restructured picture in frame memory 107, and outputs 
5 the predicted value. Motion estimation unit 109 detects the motion vector of 
the input block and the reference frame in accordance with the input picture 
block and the reference frame read from frame memory 106 to minimize a 
difference between the input block and the predicted value, i.e., the prediction 
error. Motion compensation unit 108 produces the predicted value from the 

10 reference frame stored in frame memory 106 by using the motion vector and 
the reference frame supplied from motion estimation unit 109. The predicted 
value by motion compensation unit 109 is based on the inter-frame prediction. 
Therefore, switch 122 is arranged to switch the predicted value output from 
intra-frame prediction unit 107 and the predicted value output from motion 

15 compensation unit 108 to be supplied to subtracter 121. 

The bitstream, which is the moving picture information compressed in 
the above process, is made of a variable length code, mainly including a 
conversion coefficient of each block, a quantizing parameter, a motion vector 
(to minimize the prediction error), and a reference frame (to minimize the 

20 prediction error). 

The above is the principle operation in the moving picture compression 
technique. 

Now, generally, in a digital broadcast system or a picture 
communication service, the moving picture signal is controlled about the 
25 occurring code amount, i.e., the bit rate thereof for transmission/storage. 
Then, code amount control unit 111 detects the occurring code amount 



3 



supplied from variable length encoding unit 103 and performs the two 
processes, as descried later, to control the occurring code amount. 

In the first process, code amount control unit 111 sets a target code 
amount for each frame in accordance with each picture type. When R is a 
5 code amount assigned to a frame that have not been encoded in GOP, Np 
and Nb are numbers of P-pictures and B-pictures. which have not been 
encoded in GOP, respectively, Xi, Xp, Xb are parameters, each of which 
represents frame complexity of each picture, defined by equations (1) to (3), 
and Kp and Kb are parameters in consideration of subjective picture qualities 
10 by picture type, target code amounts Ti, Tp, Tb by picture types are then given 
by equations (4) to (6). 

Xi = QixCi (1), 

Xp = QpxCp (2), 

Xb = QbxCb (3), 
15 Ti = R/(1 + NpxXp/(KpxXi) + NbxXb/(KbxXi)) (4), 

Tp = R/(Np + NbxKpxXb/(KbxXp)) (5), and 

Tb = R/(Nb + Npx KbxXp/(KpxXb)) (6). 

Where Ci, Cp, Cb are respectively occurring code amounts of I-, P-, B- 
pictures that are finally coded, and Qi, Qp, Qb are respectively average 
20 quantization step sizes of I-, P-, B-pictures that are finally coded. Incidentally, 
in the following explanations, for convenience in writing, one value among Ci, 
Cp, Cb is expressed as Ci,p,b. Also, the expression of Qi,p,b=Xi,p,b/Ti,p,b 
collectively represents the equation of Qi=Xi/Ti in relation to the l-picture, the 
equation of Qp=XpyTp in relation to the P-picture, and the equation of 
25 Qb=Xb/Tb in relation to the B-picture. 

Whenever each frame is encoded in accordance with the first process 
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and the second process, which will be described later, code amount R 
assigned to a frame that has not been encoded in GOP is updated in 
accordance with equation (7). 

R = R-Ci,p,b (7). 
5 Further, when a head picture of GOP is encoded, code amount R is 

initialized by equation (8). 

R = bit_ratex N/frame_rate + R (8). 

Where, bit_rate is a target bit rate, frame_rate is a frame rate, and N is 
the number of frames in GOP. 
10 In the second process, in order to coincide code amounts Ti, Tp, Tb 

assigned to respective frames, obtained in the first process, with actual 
occurring code amounts, quantization steps are obtained by feedback control 
in macroblock units in accordance with the virtual buffer capacity that is set by 
each picture type. 

15 First, prior to encoding of the j-th macroblock, occupation amounts in 

the virtual buffer are obtained by picture types in accordance with equation (9). 
di,p,b(j) = di,p,b(0) + BG-1) - Ti.p,bx(j-1)/MBcount (9), 
di,p,b(0) is an initial occupation amount in the vertical buffer, B(j) is an 
occurring code amount from a head to the j-th macroblock in the frame, and 
20 MBcount is the number of macroblocks in the frame. 

When encoding of each frame is finished, initial occupation amounts 
di,p,b(MBcount) in the vertical buffer by picture types are used as initial 
occujDation amounts di,p,b(0) in the vertical buffer relative to the next picture. 
Then, quantizatipn step size QQ) relative to the j-th macroblock is 
25 calculated in accordance with equation (10). 

Q(j) = Qi,p.bxdi,p.ba)x31/(10xr) (10), and 
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Qi,p.b = Xi.p.b/Ti.p.b (11). 

Where r is a parameter used to control the response speed of the 
feedback loop, which is called a reaction parameter, and is represented by 
equation (12). 

5 r = 2 X bit_rate/frame_rate (12). 

Incidentally, initial occupation amounts di,p,b(0) In the virtual buffer at 
the start of encoding are represented by equations (13) to (15). 
di(0) = 10xr/31 (13), 
dp(0) = Kpxdi(0) (14). and 

10 db(0) = Kbxdi(0) (15). 

On the other hand, a moving picture encoding scheme is proposed in 
which a multi-frame motion prediction that enables the motion prediction is 
incorporated, that is, the P-picture is predicted from not only the l-picture or 
the P-picture that is just previously encoded but also the frame that is already 
15 encoded and the B-picture is predicted from not only the l-picture or the P- 
picture that is just previously encoded but also the B-picture that is already 
encoded. In this scheme, since a high-quality frame that Is previously 
encoded is selected, and then the motion prediction can be performed degree 
of freedom in the motion prediction increases. 
20 Disclosure of Invention; 

Problems to be solved by the invention: 

However, the conventional moving picture encoding technique, as 
described above, controls the code amount assigned to each frame only by 
using the picture type and the complexity of each picture that is finally 
25 encoded, without using the degree of freedom in the motion prediction 

according to the multi-frame motion prediction. Therefore, according to the 



conventional technique, in the moving picture compression using the multi- 
frame motion prediction, moving pictures are not encoded in a high picture 
quality by using the multi-frame motion prediction effectively. Conversely, in 
the moving picture compression using the multi-frame motion prediction, a 
5 technique is desired in which moving pictures can be encoded in a high picture 
quality by using the multi-frame motion prediction effectively. 

An object of the present invention is to provide a moving picture 
encoding method of encoding a moving picture in a high picture quality by 
effectively using the multi-frame prediction. 
10 Another object of the present invention is to provide a moving picture 

encoding apparatus for encoding a moving picture in a high picture quality by 
effectively using the multi-frame prediction. 
Means for Solving the Problems: 

The object of the present invention is attained by a moving picture 
15 encoding method for performing a multi-frame motion prediction with reference 
to a plurality of picture frames, wherein the method comprises including, in 
reference frames used for the multi-frame motion prediction, a frame that has 
been encoded in a higher picture quality than the other frames of the same 
picture type. 

20 Another object of the present invention is attained by a moving picture 

encoding apparatus for performing a multi-frame motion prediction with 
reference to a plurality of picture frames, the apparatus being characterized in 
that it includes selection means for selecting at least one reference frame from 
a plurality of reference frames of the same picture type, and encoding means 

25 for encoding the selected reference frame in a higher picture quality than the 
other reference frames of the same picture type. 
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According to the present invention, in the moving picture compression 
using the multi-frame motion prediction, not the picture type and the 
complexity of each picture that is finally encoded are simply used, but code 
amount control is performed so as to improve the effects of the motion 
5 prediction of the whole scene by encoding a frame having high priority as a 
reference frame in a high picture quality in consideration of the relationship 
between the subject frame to be encoded and the reference frame in the multi- 
frame motion prediction. With this arrangement, according to the present 
invention, in the moving picture compression using the degree of freedom of 

10 the motion prediction by the multi-frame motion prediction, the moving picture 
encoding method in the higher picture quality can be carried out. 

Also, in accordance with the scene of the moving picture, frames to be 
encoded in a higher picture quality are arranged at constant frame intervals or 
the frame interval of frames to be encoded in a higher picture quality are 

15 adaptively changed in accordance with differential information and motion 
information between the reference frame and the subject frame to be 
encoded, thereby precisely selecting the reference frame to be encoded in a 
higher picture quality with priority, and thus the code amount control for 
moving pictures in the higher picture quality can be provided. Accordingly, the 

20 moving pictures can be encoded in a still higher picture quality. 

Also, the more code amount is assigned to the B-picture frame with 
high priority as a reference frame, thereby providing code amount control for 
moving pictures in a higher picture quality. Accordingly, since the effects of 
the motion prediction of the whole sequence can be improved, moving pictures 

25 can be encoded in a higher picture quality. 

Brief Description of the Drawings: 
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Fig. 1 is a block diagram showing a configuration of a conventional 
typical moving picture encoding apparatus. 

Fig. 2 is a view for explaining a moving picture encoding process 
according to tlie present invention. 
5 Fig. 3 is a view for explaining the moving picture encoding process 

according to the present invention. 

Fig. 4 is a block diagram showing a configuration of a moving picture 
encoding apparatus according to a first embodiment of the present invention. 

Fig. 5 is a block diagram showing a configuration of a picture quality 
10 control unit. 

Fig. 6 is a flowchart showing a picture quality determination process. 
Fig. 7 is a view for explaining a frame to be encoded in a high picture 
quality. 

Fig. 8 is a flowchart showing a picture quality control counter process. 
15 Fig. 9 is a block diagram showing a configuration of a code amount 

control unit. 

Fig. 1 0 is a flowchart showing a process for assigning a code amount to 
a frame. 

Fig. 1 1 is a flowchart showing a process for updating a quantizing 
20 parameter. 

Fig. 12 is a block diagram showing a configuration of a moving picture 
encoding apparatus according to a second embodiment of the present 
invention. 

Fig. 13 is a block diagram showing a configuration of a moving picture 
25 analysis unit. 

Fig. 14 is a flowchart showing a motion detection process. 
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Fig. 15 is a block diagram showing a configuration of a picture quality 
control unit. 

Fig. 16 is a flowchart showing a picture quality determination process. 
Fig. 17A is a view for explaining an optimal reference frame. 
5 Fig. 17B is a view for explaining the optimal reference frame. 

Fig. 18A is a view for explaining the difficulty of the inter-frame 
prediction. 

Fig. 18B is a view for explaining the difficulty of the inter-frame 
prediction. 

10 Fig. 19 is a flowchart showing a process for determining a frame 

encoded in a high picture quality. 

Fig. 20 is a view for explaining a frame to be encoded in a high picture 
quality. 

Fig. 21 is a flowchart showing a picture quality determination process. 
15 Fig. 22 is a view for explaining a frame to be encoded in a high picture 

quality. 

Fig. 23 is a flowchart showing a process for assigning a code amount to 
a frame. 

Fig. 24 is a flowchart showing a picture quality determination process. 
20 Fig. 25 is a typical block diagram of an information processing system 

installed with the moving picture encoding apparatus according to the present 
invention. 

Fig. 26 is a block diagram showing an example of a receiver to which a 
moving picture bitstream generated according to the present invention is 
25 supplied. 

Fig. 27 is a flowchart showing a process for generating a reference 
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frame picture quality fluctuation flag. 

Best Mode for Carrying Out the Invention: 

In the present invention, as described above, an encode amount is 
controlled in a manner that effects of motion prediction of whole scene are 
5 improved, in consideration of the relationship between a subject frame to be 
encoded and a reference frame in the multi-frame motion prediction, by 
encoding a frame with a high priority as the reference frame in a high picture 
quality. 

In the following description, "scene" means any number of continuous 

10 frames. "A frame is encoded in a high picture quality" means that more code 
amount is assigned to the frame, or that the quantization step size of the 
frame is made small, or that coding distortion in the frame is reduced. 

Hereinafter, the basic principle of the present invention is explained as 
compared with the conventional art. 

15 According to the present invention, in the moving picture encoding 

method using the multi-frame motion prediction, picture qualities of frames of 
the same picture type are not even but variable, thereby enhancing the effects 
of the motion prediction of the whole scene to encode a moving picture in a 
high picture quality. On the other hand, in the conventional moving picture 

20 encoding method using the motion prediction method, because the motion 
prediction is performed only by the picture that has been just encoded, only 
the encoding can be performed in which the picture qualities of frames of the 
same picture type are kept even. As its reason, when the picture qualities in 
the same scene are varied, a difference between frames increases in a frame 

25 that refers to another frame in the lowered picture quality and the effects of the 
motion prediction are lowered. Therefore, the encoding efficiency of the whole 
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scene is lowered, and thus the picture quality of the moving picture that is 
encoded is lowered. 

Next, this problem is explained with reference to Fig. 2. 
As indicated by solid lines in Fig. 2, in the conventional motion 
5 prediction, each P-picture refers to only the I- or P-picture that has been just 
encoded. Even if P2. P4 frames are tried to be encoded in a higher quality 
than Pi, P3 frames, Pi frame is encoded in the low picture quality, and 
therefore the effects of the motion prediction in P2 frame that refers to Pi 
frame are significantly lowered and the picture quality in P2 frame is also 

10 significantly lowered. The picture quality in P3 frame that refers to P2 frame is 
also significantly lowered, and because the picture quality lowering propagates 
in the following frames, the picture quality of the whole scene is lowered. 
Therefore, in the method using the conventional motion prediction, the picture 
quality of pictures of the same picture type must be kept even. 

15 However, like the coding method to which the present invention is 

applied, there is a multi-frame motion prediction that can perform a motion 
prediction, by referring to not only the frame that has been just encoded, but 
also the frames that have been already encoded. In the case of Fig. 2 
according to the multi-frame motion prediction, predictions indicated by dot 

20 lines can be performed in addition to the conventional method. In this case, a 
frame to be referred can be selected by an area or by a block in the frame. 

With this multi-frame prediction arrangement, it is possible to perform 
the encoding that varies the picture quality in spite of the same picture type. 
The present invention is explained with reference to Fig. 2. 

25 In the multi-frame motion prediction shown in Fig. 2, each P-picture 

refers to a plurality of I- or P-pictures that have been just encoded. In this 
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example, two I- or P-pictures are referred. Even if Pi frame is encoded in the 
low quality when P2, P4 frames are tried to be encoded in a higher quality than 
Pi, P3 frames, in P2 frame that refers to not only Pi frame but also \^ frame, 
the effects of the motion prediction are not significantly lowered. Therefore, P2 
5 frame is encoded in a higher quality. 

Also, the effects of the motion prediction in P3 frame that refers to P2 
frame encoded in the high picture quality are enhanced, and P3 frame is 
encoded in a higher quality than the conventional art. Further, since P4 frame 
refers to P2, P3 frames that are encoded in the high picture quality, P4 frame is 

10 encoded in a higher quality. In the following frames, the effects of the motion 
prediction are enhanced and moving pictures can be encoded in a high picture 
quality, similarly to the above frames. 

With this arrangement, the effects of the motion prediction are 
enhanced rather than the conventional art, and moving pictures can be 

15 encoded in a high picture quality. Also, the moving pictures that are encoded 
according to the present invention subjectively provide the higher picture 
quality than the conventional art, because the frames that are encoded in the 
high picture quality are periodically displayed when being decoded, and by 
persistence of vision, i.e., by the human visual feature. 

20 In the above description, the example using only I- and P-picture is 

shown, however, the present invention is available to a case of using P-, 
and B-pictures are used. With reference to Fig. 3, explanations are given of 
the advantages according to the present invention in the case where I-, P-, 
and B-pictures exist, in comparison with the conventional art. Incidentally, the 

25 basic concept relative to P-picture is similar to that shown in Fig. 2, and 

therefore explanations are given below while paying attention to only B-picture. 
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In the conventional art, the B-picture frame is a frame not to be 
referred, i.e., a frame that is not referred from another frame. Therefore, when 
B-picture frames are continuous, the effects of the motion prediction in 
respective B-pictures, which are continuous, are equal. Therefore, the same 
5 code amount is assigned to the respective B-frames, which are continuous, 
and the B-frames are encoded in the same picture quality. As the reason in 
that the effects of the motion prediction of respective B-pictures, which are 
continuous, are made equal, when attention is given to Bi and B2 frames in 
the exarhple of Fig. 3, distances from lo, P3 frames to Bi frame are respectively 

10 one and two, and distances from lo, P3 frames to B2 frame are respectively two 
and one, and both totals of frame distances for determining the effects of the 
motion prediction are three. 

However, in the moving picture encoding that uses the multi-frame 
motion prediction, a B-picture can also refer to another B-picture that is 

15 previously encoded. Therefore, the effects of the motion prediction of 

continuous B-pictures are variable. When an attention is given to Bi and B2 
frames in Fig. 3, B2 frame can perform the motion prediction that refers to not 
only lo frame but also Bi frame having a short inter-frame distance, and B2 
frame can provide higher performance of the motion prediction than Bi frame, 

20 as is apparent. Therefore, even if the same code amount is not assigned to 
Bi and B2 frames, Bi and B2 frames can be encoded in the same picture 
quality. 

With this technique, according to the present invention, in the moving 
picture encoding method using the multi-frame motion prediction, a code 
25 amount is distributed to a leading reference B-frame by priority in order of 
encoding in continuous B-frames, thereby performing the moving picture 
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encoding in a high picture quality. Since the code amount is assigned to Bi 
and B2 frames in Fig. 3 by priority, Bi frame is encoded in the higher picture 
quality than the conventional art and 82 frame, which refers to Bi frame, is 
encoded in the higher picture quality, can be encoded in a higher picture 
5 quality. In other words, by varying picture qualities of B-picture frames, the 
effects of the whole motion prediction are enhanced and moving pictures can 
be encoded in a higher picture quality. 

Explanations are given below of concrete embodiments of the present 
invention. 

10 A moving picture encoding apparatus according to the first embodiment 

of the present invention, shown in Fig. 4, further includes picture quality control 
unit 1 12 in addition to the moving picture encoding apparatus shown in Fig. 1 , 
and code amount control unit 111 determines an assignment code amount 
that corresponds to a reference frame, based on picture quality control 

15 information supplied from picture quality control unit 112. In Fig. 4, the same 
reference numerals are applied to the same constitutional elements in Fig. 1 , 
and detailed explanations thereof are omitted. Incidentally, in the first 
embodiment, the operation is explained when l-picture and P-picture are used 
as picture types used to encode frames. Hereinafter, explanations are given 

20 of picture quality control unit 112 and code amount control unit 111, which are 
distinctive of the moving picture encoding apparatus according to the first 
embodiment. 

As shown in Fig. 5, picture quality control unit 112 includes picture 
quality determination unit 1121 and picture quality control counter 1122. The 
25 picture quality control inforhnation supplied from picture quality control unit 112 
includes high picture quality flag HQ_flg, number of remaining high picture 
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quality frames R_HQ_num, and remaining higli picture quality frame number 
R_HQ_frame_num. Picture quality determination unit 1121 in picture quality 
control unit 112 calculates high picture quality flag HQ_flg, number of high 
picture quality frames R_HQ_num. and high picture quality frame number 
5 R_HQ_frame_num, and supplies high picture quality flag HQ_flg to code 
amount control unit 111 and supplies number of high picture quality frames 
HQ_num and high picture quality frame number HQ_frame_num to picture 
quality control counter 1 122. 

Hereinafter, the operation of picture quality determination unit 1121 

10 according to the first embodiment is explained with reference to Fig. 6. In the 
following explanation, a frame interval between mutual frames to be encoded 
in a high picture quality is represented by S, a frame number that has been 
just determined to be encoded in a high picture quality is represented by 
prev_hq_num, and the number of frames that have been determined to be in a 

15 high picture quality is represented by i. Additionally, S<MAX_REF is satisfied, 
where an interval between a subject frame to be encoded and the previous- 
most frame that can be referred by the multi-frame prediction is represented 
by MAX_REF. Both the initial values of prev_hq_num and i are 0. 

Incidentally, in the first embodiment, frame interval S that is encoded in 

20 a high picture quality may be changed in accordance with a motion speed or a 
frame rate after compression by a period, for example, by GOP or by a scene. 
By selecting frame interval S by GOP or by scene so as to be suitable thereto, 
the encoding efficacy according to the present invention is further improved. 
In step SI 01, picture quality determination unit 1121 calculates number 

25 of high picture quality frames HQ_num by using equation (16) based on frame 
interval S and number N of frames in GOP. 
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HQ_num = (N/S) - 1 (16). 

Then, in step S102, picture quality determination unit 1121 determines 
whether or not number i of frames that have been determined to be in a high 
picture quality is smaller than number of high picture quality frames HQ_num. 
5 When smaller, step SI 03 is executed, and when not smaller, the process is 
finished. 

In step SI 03, picture quality determination unit 1121 calculates high 

picture quality frame number HQ_frame_num[i] by using equation (17). 

Further, number i of frames that have been determined to be in a high picture 
10 quality is incremented by 1, frame number prev_hq_num that has been just 

determined to be encoded in a high picture quality is updated by equation (18), 

and step SI 02 is executed. 

HQ_frame_num[i] = prev_hq_num + S (17), and 

prev_hq_num = HQ_frame_num[i] (18). 
15 After the above processes, picture quality determination unit 1 121 turns 

HQ_Jlag on when HQ_num is not less than 1 and turns HQ_flag off when less 

than 1. 

In the moving picture encoding apparatus of the present embodiment, 
according to the process shown in Fig. 6, the reference frame to be referred 

20 when another frame is encoded, is encoded in a high picture quality at interval 
S. Because S is not more than interval MAX_REF of previous-most frame that 
can be referred in the multi-frame prediction, the motion prediction can be 
necessarily performed in all subject frames to be encoded while referring to 
frames that are encoded in the high picture quality. 

25 As an example. Fig. 7 shows frames to be encoded in the high picture 

quality according to the method of the present embodiment, in the case of 
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N=10, MAX_REF=3, and S=2. In this case, each frame of P2. P4. Pe, Ps is 
encoded in a high picture quality. 

Picture quality control counter 1 122 calculates number of remaining 
high picture quality frames R_HQ_num and remaining high picture quality 
5 frame number R_HQ_frame_num in accordance with frame number 

coding_frame_num supplied from code amount control unit 111 and number of 
high picture quality frames HQ_num and high quality frame number 
HQ_frame_num[HQ_num] supplied from picture quality determination unit 
1 122, and outputs them to code amount control unit 111. Picture quality 

10 control counter 1 122 operates only when number of high picture quality frames 
HQ_num is not less than 1 . Here, the frame number of l-picture at the head of 
GOP is represented by frame_num_l. high picture quality frame counter 
HQ_frame_count is set to 0, and the operation of picture quality control 
counter 1 122 is explained with reference to Fig. 8. 

15 In step S201, picture quality control counter 1 122 adds frame_numj to 

high picture quality frame number HQ_frame_num[HQ_frame_count] supplied 
from picture quality determination unit 1 121. This operation is performed to 
synchronize frame number coding_frame_num in being encoded, supplied 
from code amount control unit 111, with the frame number to be encoded in a 

20 high picture quality. 

In step S202, picture quality control counter 1 122 takes 
HQ_frame_num[HQ_frame_count] as remaining high picture quality frame 
number R_HQ_frame_num and outputs (HQ_num - HQ_frame_count) as 
number of remaining high picture quality frames R_HQ_num. Then, in step 

25 S203, picture quality control counter 1 122 detemriines whether or not frame 
number coding_frame_num is synchronized with hjgh picture quality frame 
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number HQ_frame_nunn[HQ_frame_count]. When they are synchronized, the 
process is advanced to step S204, and when they are not synchronized, the 
process is returned to step S202. 

In step S 204, picture quality control counter 1 122 increments 
5 HQ_frame_count by 1 and determines whether or not HQ_frame_count is 
smaller than HQ_num. When HQ_frame_count is smaller than HQ_num, the 
process is advanced to step S201, and when not smaller, the process is 
finished. 

As described above, picture quality control counter 1 122 outputs 
10 remaining high picture quality frame number R_HQ_frame_num and number 
of remaining high picture quality frames R_HQ_num to code amount control 
unit 111. 

Code amount control unit 1 1 1 , as shown in Fig. 9, frame code amount 
assignment unit 1111 and quantizing parameter update unit 1112. Code 

15 amount control unit 1 1 1 in the present embodiment is different from the code 
amount control unit in the conventional moving picture encoding apparatus 
shown in Fig. 1 in the operation of frame code amount assignment unit 1111. 

Frame code amount assignment unit 1 1 1 in the present embodiment 
calculates a code amount assigned to each frame by using the picture quality 

20 control information (high picture quality flag HQ_flg, number of remaining high 
picture quality frames R_HQ_num, and remaining high picture quality frame 
number R_HQ_frame_num) supplied from picture quality control unit 112 and 
supplies the code amount to quantizing parameter update unit 1112. 
Quantizing parameter update unit 1112 calculates a quantizing parameter by 

25 using the frame assignment code amount supplied from frame code amount 
assignment unit 1111 and the occurring code amount supplied from buffer 110 
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and supplies the quantizing parameter to variable length encoding unit 103. 

The operation of frame code amount assignment unit 1111 is explained 
below. In the following explanations, it is assumed that target code amounts 
Ti, Tp are target code amounts by picture type. R is a code amount assigned 
5 to frames that have not been encoded in GOP. Np is the number of P-pictures 
that have not been encoded in GOP, Xi is screen complexity of l-picture that is 
finally encoded, Xp is screen complexity of P-picture that is finally encoded, 
and Kp is a parameter by picture type in consideration of the subjective picture 
quality. 

10 Figure 10 shows the operation of frame code assignment unit 1111 

according to the first embodiment. 

First, in step S301, frame code amount assignment unit 1111 
determines whether high picture quality flag HQ_flg supplied from picture 
quality control unit 1 12 is turned on or off. When high picture quality flag 
15 HQ_flg is turned on, the process is advanced to step S302, and when off, the 
process is advanced to step S307. 

In step S302, it is determined whether or not the frame to be being 
encoded is l-picture. When the subject frame to be encoded is l-picture. the 
process is advanced to step S303, and when not l-picture, the process is 
20 advanced to step S304. 

In step S303, frame code amount assignment unit 1111 calculates code 
amount Ti for l-picture that is being coded, by equation (19) and finishes the 
frame code amount assignment. 

Ti = R/(1 + NpxXp/Xi) + additionaLTi (19). 
25 additionaLTi = residu1_bit1 xXi/Xgop2 (20). 
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residu_bit1 = (margin_ratioxRx(Np - R_HQ_num)xXp)/(KpxXgop1) 

(21). 

Xgop1 = Xi + NpxXp/Kp (22), and 

Xgop2 = Xi + R_HQ_numxXp/Kp (23). 
5 Where. margin_ratio is a number not more tlian 1 . 

In this case, since the number of bits more then the conventional art by 
additionaLTi is assigned to the l-frame, the picture quality of this frame is 
improved. Accordingly, the effects of the frame motion prediction that refers to 
this frame are improved. 
10 In step S304, frame code amount assignment unit 1111 determines 

whether or not frame number coding_frame_num of P-picture that is being 
encoded is synchronized with remaining high picture quality frame number 
R_HQ_frame_num supplied from picture quality control unit 112. When they 
are synchronized, the process Is advanced to step S305, and when not 
15 synchronized, the process is advanced to step S306. 

In step S305, frame code amount assignment unit 1111 calculates the 
code amount for P-picture that is being encoded by equation (24) and finishes 
the frame code amount assignment. 

Tp = R/Np + additional_Tp (24), and 

20 additionaLTp = (margin_ratiox Rx (Np - R_HQ_rum))/(Npx R_HQ_num) 

(25). 

In this case, since the number of bits more then the conventional art by 
additionaLTp is assigned to P-frame. the picture quality of this frame is 
improved. Accordingly, the effects of the frame motion prediction that refers to 
25 this frame are also improved. 

In step S306. frame code amount assignnrient unit 1111 calculates the 
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code amount for P-picture that is being encoded by equation (26) and finishes 
the frame code amount assignment 

Tp = (1 - margin_ratio)x R/Np (26). 

In this case, since the code amount assigned to the frame is decreased 
5 by margin_ratio, it is considered that the picture quality of this frame is 

lowered. However, since the motion prediction can be performed by control of 
picture quality control unit 112 while referring to the frames encoded in the 
high picture quality, the picture quality can be prevented from lowering by 
improvement of motion prediction performance, even if the assigned code 
10 amount is small. 

In step S307, frame code amount assignment unit 1111 determines 
whether or not the frame that is being encoded is l-picture. When the subject 
frame to be encoded is l-plcture, the process is advanced to step S308. and 
when not l-picture, the process is advanced to step S309. 
15 In step S308, frame code amount assignment unit 1111 calculates the 

code amount for l-picture that is being encoded by equation (27) and finishes 
the frame code amount assignment. 

Ti = R/(1 + NpxXp/Xi) (27). 

Similarly, in step S309, frame code amount assignment unit 1111 
20 calculates the code amount for P-picture that is being encoded by equation 
(28) and finishes the frame code amount assignment. 
Tp = R/Np (28). 

According to the code amount assignment, as described above, all 
frames can refer to the frames encoded in the high picture quality by using the 
25 degree of freedom of the motion prediction in the multi-frame motion 

prediction. With this arrangement, according to the present embodiment, 



22 



since the effects of the motion prediction of the whole scene are improved, 
moving pictures can be encoded in a high picture quality. 

Quantizing parameter update unit 1112 performs feedback control to a 
quantizing parameter by macroblock, based on the virtual buffer capacity that 
5 is set for each picture type, in order to match code amounts Ti, Tp that are 
assigned to respective frames and are obtained in frame code amount 
assignment unit 1111 with the actual occurring code amounts. Figure 1 1 is a 
flowchart of updating the quantizing parameter. 

First, in step S401, quantizing parameter update unit 1112 calculates 
10 an occupation amount of the virtual buffer by picture type with equation (29) 
prior to encoding of the j-th macroblock. 

cli,pO) = di,p(0) + BO-1) - Ti,pxO-1)/MBcount (29). 

Where di,p(0) is an initial occupation amount in the virtual buffer by 
picture type, B(j) is an occurring code amount from the head of the frame to 
15 the j-th macroblock, and MBcount is the number of macroblocks in the frame. 
Needless to say. the initial occupation amount in the virtual buffer for l-picture 
is di(0) and that for P-picture is dp(0). 

At the completion of encoding each frame, initial occupation amounts 
di,p(MBcount) in the virtual buffer by picture type are used as initial occupation 
20 amounts di,p(0) in the virtual buffer for the next picture. 

Then, in step S402, quantizing parameter update unit 1112 calculates 
the quantization step size relative to the j-th macroblock by equation (30). 

Qstep = Qi,pxdi,pG)x31/(10xr) (30), and 

Qi,p = Xi.p/Ti,P (31). 
25 With control by frame code amount assignment unit 1111, assignment 

cpde amounts Ti.p of frames to be encoded in the high picture quality become 
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larger than those of the conventional art. Therefore, as is apparent from 
equation (31), quantization step sizes Qi,p of high picture quality frames are 
small and the frames are encoded in the high picture quality. Accordingly, the 
average quantizing parameter of the frames of the bitstream to be output 
5 provides small values at frame intervals of S. 

Here, r Is a parameter, called a reaction parameter, for controlling the 
response speed of the feedback loop, and is given by equation (32). 

r = 2 X bit_rate/f rame_rate (32). 

Incidentally, at start of encoding, initial occupation amounts di,p(0) of 
10 the virtual buffer for each picture type are given by equations (33). (34). 

di(0)= 10xr/31 (33), and 

dp(0) = Kpxdl(0) (34). 

Then, in step S 403, quantizing parameter update unit 1112 detects 
quantizing parameter Q corresponding to quantization step size Qstep from a 
15 quantization table. When there is no quantization step size Qstep in the 
quantization table, quantizing parameter Q of the quantization step size 
closest to quantization step size Qstep is output. 

As described above, the moving picture encoding apparatus according 
to the first embodiment encodes a frame having high priority as a reference 
20 frame in a high picture quality, whereby the code amount control is performed 
so that the effects of the motion prediction of the whole scene are improved. 

Next, explanations are given of a moving picture encoding apparatus 
according to the second embodiment of the present invention. A moving 
picture encoding apparatus according to the second embodiment of the 
25 present invention, shown in Fig. 12, further includes moving picture analysis 
unit 1 13 which performs analysis of a moving picture in addition to the moving 
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picture encoding apparatus according to the first embodiment shown in Fig. 4. 
Moving picture analysis unit 113 outputs frame differential information and 
frame motion information, and picture quality control unit 112 generates 
picture quality control information by further using the frame differential 
5 information and the frame motion information. Therefore, in the moving 
^ picture encoding apparatus according to the second embodiment, moving 
picture analysis unit 113 and picture quality control unit 112 are intensively 
explained. Incidentally, in the second embodiment, explanations are given of 
a case where l-picture and P-picture are used as picture types to encode 
1 0 frames. 

Moving picture analysis unit 113 calculates the frame differential 
information and the frame motion information from the input picture, supplies 
the calculated information to picture quality control unit 112, and includes input 
frame buffer 1131 and motion detection unit 1 132, as shown in Fig. 13. 
15 Input frame buffer 1 131 foresees N-pieces of input moving picture 

frames, allocates numbers to respective frames in ascending order, and is 
stored with the frames. The number of frames to be foreseen is arbitrary, and, 
for example, it can be considered that the GOP interval used in MPEG is used 
as N. 

20 Motion detection unit 1 132 calculates the frame motion information and 

frame differential information of each frame in accordance with the frames 
stored in input frarhe buffer 1 1 32. The operation of motion detection unit 1 1 32 
is explained below with reference to Fig. 14. Motion detection unit 1132 
performs each step shown in Fig. 14 for each frame, which is foreseen. 

25 First, in step S501 , motion detection unit 1 132 divides the frame into 

blocks of wxh in size smaller than the frame size of WxH, and, in step S 502, 
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calculates block motion Information MVX, MVY and block differential 
information D of the blocks to form the frame in accordance with the following 
process. 

A pixel value at coordinate (x, y) in the cur-th frame is set as F(cur, x, y), 
5 the k-th divided block in the cur-th frame is set as B(cur, k), and a coordinate 
at the upper-left corner of B(cur, k) in the frame is set as (bx(cur, k), by(cur, 
k)). 

Block motion information MVX(cur, k, ref), MVY(cur, k, ref) and 
differential information D(cur, k, ref) of block (cur, k) in the cur-th frame, 
10 relative to a previous frame from that frame by ref-pieces, i.e., (cur - ref)-th 

frame, are given by mvx, mvy that minimize diff(ref, mvx, mvy) in equation (35) 
and minimized diff(ref, MVX(cur, k, ref), MVY(cur, k, ref)). 

/=tv-i. y=/)-i 

diff{ref, mvx, mvy) 2l (^^'' _ pixel (/, J) -ref _ pixel (ref, i + mvx, j + mvy)) 

(35), 

curj)ixel(i, j) = F(cur, bx(cur, k) + i, by(cur, k) + j) (36), and 

15 ref_pixel(ref, i ,j) = F(cur - ref, bx(cur, k) + i, by(cur, k+j)) (37). 

Then, in step S503, motion detection unit 1 132 calculates the frame 
motion information and the frame differential information from block motion 
information MVX, MVY and block differential information D. 

When the number of blocks in the frame is set as block_num, frame 
20 motion information FMVj(i) and frame differential information FDj(i) of the i-th 
frame relative to the (i-j)-th frame are given by equations (38) and (39). 

biock num-l 

FMVj(i)= ^{abs{MVX{Un,j)+MVY{},nj))) (38), 

blodc nuw-1 V ^ ^ 

FDm= XD{Un,j) (39). 
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Motion detection unit 11 32 performs the above-mentioned process for 
each frame i (1 <I<N-1) and each reference frame ref (1 <ref<MAX_REF) and 
calculates the frame differential information and the frame motion information 
between all the input frames and the reference frames. 
5 Picture quality control unit 1 12, as shown in Fig. 15. includes picture 

quality determination unit 1121 and picture quality control counter 1 122. 
Picture quality control information supplied from picture quality control unit 112 
includes a high picture quality flag, the number of remaining high picture 
quality frames, and a remaining high picture quality frame number. Picture 
10 quality control unit 1 12 of the present embodiment is different from picture 
quality control unit 1 12 of the first embodiment, as described above, in the 
operation of picture quality determination unit 1 121. Therefore, the operation 
of picture quality determination unit 1121 according to the second embodiment 
is explained below. 

15 Picture quality determination unit 1121 calculates the high picture 

quality flag, the number of high picture quality frames, and the high picture 
quality frame number in accordance with the frame differential information and 
the frame motion information supplied from moving picture analysis unit 112. 
The high picture quality flag is output to code amount control unit 111, and the 

20 number of high picture quality frames and the high picture quality frame 

number are output to picture quality control counter 1 122. The operation of 
picture quality determination unit 1121 according to the present embodiment is 
explained below with reference to Fig. 16. 

First, in step S60i , picture quality determination unit 1 121 calculates 

25 inter-frame cost IFC(i) of each frame i and reference frame j by using equation 
(40). a is a value depiBnding on the average quantizing parameter of the 
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frame. 

IFCj(i) = FDj(i) + a X FMVj(i) (40), 

MiN.iFcm = rnki r 

Then, in step S602, picture quality determination unit 1121 calculates 
5 minimum inter-frame cost MINJFC(i) (1 <i<N-1) by using equation (41) and 
obtains best reference frame BEST_REF(i) corresponding to MINJFC(i). 
Here, MAX_REF is an interval between the subject frame to be encoded and 
the previous-most frame that can be referred in the multi-frame prediction. 
In step S603, picture quality determination unit 1121 determines 

1 0 whether or not BEST_REF(j) (1 < i < N-1 ) of all frames are 1 . As shown in Fig. 
17A, when BEST_REF(i) of all frames are 1, a continuous scene is predicted 
without momentary picture fluctuations, such as a scene change and a flash, 
in N-pieces of analysis sections. On the other hand, as shown in Fig. 178, 
when there is a frames of which BEST_REF(i) are not 1 , a discontinuous 

15 scene is predicted with a scene change and a flash in N-pieces of analysis 

sections. Therefore, when all BEST_REF(i) are 1 in step S603, the process is 
advanced to step S604, and when all BEST_REF(i) are not 1, the high picture 
quality flag is turned off and the number of high picture quality frames is set to 
0, and then the process is finished. 

20 In step S604, picture quality control unit 112 calculates average inter- 

frame cost AVERAGEJFC, inter-frame cost upper limit IFC_UPPER_LIMIT. 
and inter-frame cost lower limit IFC_LOWER_LIMIT in accordance with 
MINJFC(i)(==IFC1(i)) of all frames by equations (42) to (44). Here, 
margin_ratio is a value not more than 1 . 



28 



AVERAGE JFC^'^MINJFC^^^^^ (42), 

" ■ . 7-1- 

IFC_UPPER_LIMIT = (1 + margin_ratio)x AVERAGE JFC (43), and 
IFC_LOWER_LIMIT = (1 - margin_ratio)xAVERAGEJFC (44). 
Then, in step S605, picture quality control unit 112 analyzes whether or 
5 not MINJFC(i) (1 <i<N-1) of all frames are within a range between 

IFC^LOWER^LIMIT and IFC_UPPER_LIMIT. When MINJFC(i) of all N-1 
pieces of picture frames are within the above-mentioned range, as shown in 
Fig. 18A, the difficulty of the inter-frame prediction is stable, and when not 
within the range, the difficulty is unstable, as shown in Fig. 18B. 

10 Therefore, picture quality control unit 112 turns the high picture quality 

flag on and advances the process to step S606 when MINJFC(i) of all frames 
are within the above-mentioned range in step S605. Otherwise, picture quality 
control unit 112 turns the high picture quality flag off, sets the number of high 
picture quality frames to 0, and terminates the following process, 

15 In step S606, picture quality control unit 1 12 adaptively calculates frame 

interval S to be encoded in a high picture quality within SS in accordance with 
the process, which will be described later, while maximum high picture quality 
frame interval set by GOP or scene is represented by SS. In this description, 
maximum high picture quality frame interval SS is a value not more than the 

20 interval between the subject frame to be encoded and the previous-most 
frame that can be referred in the multi-frame prediction and is suitably set in 
accordance with the motion speed and the frame rate after moving picture 
compression. 

The number of continuous frames not over IFC_UPPER_LIMIT is 
25 represented by s_frame, frame number that has been just determined to be 
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encoded in a high picture quality is represented by prev_fnum, and frame 
number in picture quality determination process is represented by i, and an 
example of the calculation method for the number of high picture quality 
frames HQ_num and high picture quality frame number HQ_frame_num[N-1] 
5 is shown with reference to Fig. 19. In this example, the initial value of s_frame 
is 1, the initial value of prev_fnum is 0, the initial value of i is 2, and the initial 
value of HQ_num is 0. 

Figure 19 shows the process in step S606 in further details. 

First, in step S701, picture quality control unit 112 determines whether 
10 or not lFCs_frame+1(i) is larger than IFC_UPPER_LIMIT, advances the 
process to step S702 when IFCs__frame+1(i) is larger than 
IFC_UPPER_LIMIT, and advances the process to step S707 when not larger. 

In step S702, picture quality control unit 112 obtains the high picture 
quality frame number in accordance with equation (45), updates prev_fnum, 
15 HQ_num, and s_frame in accordance with equations (46) to (48), and 
advances the process to step S706. 

HQ_frame_num[hq_num] = prev_fnum + s_frame (45), 

prev_fnum = HQ_frame_num[hq_num] (46), 

HQ_fnum = HQ_fnum + 1 (47), and 

20 s^frame =1 (48). 

In step S703, picture quality control unit 112 determines whether or not 
s_frame is equal to SS-1 , advances the process to step S704 when equal, and 
advances the process to step S705 when not equal. 

In step S704, picture quality control unit 112 obtains the high picture 
25 quality frame number in accordance with equation (45) while setting 

s_frame=SS, updates prev_fnum, HQ_num, and s_frame in accordance with 
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equations (46) to (48), and increments frame number i by 1. Then, the 
process is advanced to step S706. 

In step S705, picture quality control unit 112 increments s_frame by 1, 
and advances the process to step S706. 
5 In step S706, picture quality control unit 112 increments frame number i 

to be processed by 1 , determines whether or not 1 is smaller than number N of 
foreseeing frames, and advances the process to step S701 when small and 
finishes the process when not small. 

With the above-mentioned process, while all the subject frames to be 

10 encoded refer to the frames that are coded in the high picture quality without 
exception, the motion prediction becomes possible. Incidentally, when 
HQ_num obtained by the above-mentioned picture quality determination 
process is equal to N-1 , namely, when it is determined that all frames are 
encoded in the high picture quality, HQ_num is reset to 0, the high picture 

15 quality flag is turned off, and the number of high picture quality frames is set to 
0. 

Figure 20 shows an example in that the above-mentioned high picture 
quality encoding frame determination is executed when N=15, REF_NUM=3, 
and the leading frame, that is, 0-th frame, is l-picture. According to the 

20 present embodiment, frame interval S to be coded in the high picture quality is 
updated adaptively. 

As described above, according to the second embodiment, moving 
picture analysis unit 113 is arranged, thereby selecting the frame to be 
encoded in the high picture quality with accuracy, in consideration of 

25 momentary picture fluctuations, such as a scene change and a flash, in GOP, 
i.e., rapid reduction of pixel correlation between continuous frames and the 



31 



effects of large fluctuations in encoding difficulty of the scene. Accordingly, 
according to the present embodiment, moving pictures can be encoded a still 
higher picture quality. 

Also, there is no limitation on the above-mentioned method of selecting 
5 frames to be encoded in the high picture quality. For example, a simple 

embodiment using inter-frame cost IFC(i) is available. For example, when the 
interval between the subject frame to be encoded and the previous-most 
frame that can be referred in the multi-frame prediction is MAX_REF, high 
picture quality frame number R_HQ_frame_num that has been just encoded in 

10 the high picture quality is regarded as one of reference frames and, in frame 
number (R_HQ_frame_num + m) where m<MAX_REF, a frame of which inter- 
frame cost IFC(i) relative to the high picture quality reference frame is smaller 
than a predetermined threshold and which corresponds to maximum m can be 
set as the next high picture quality frame. 

15 Next, explanations are given of a moving picture encoding apparatus 

according to the third embodiment of the present invention. In the third 
embodiment, l-picture, P-picture, and B-picture are used as picture types used 
to encode frames. The moving picture encoding apparatus of the third 
embodiment is configured similarly to the moving picture encoding apparatus 

20 of the first embodiment, however, the third embodiment is different from the 
first embodiment in the operations of picture quality control unit 112 and code 
amount control unit 111, because encoding is performed by using B-picture. 
Hereinafter, explanations are given of the operations of picture quality control 
unit 112 and code amount control unit 111 in to the third embodiment. 

25 Picture quality control unit 112 includes picture quality determination 

unit 1 121 and picture quality control counter 1 122i sihiilarly to the first 
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embodiment. Picture quality control unit 112 supplies code amount control 
unit 111 with the high picture quality flag, the number of remaining high picture 
quality frames, and the high picture quality frame number, as the picture 
quality control information. The number of remaining high picture quality 
5 frames and the high picture quality frame number relate to P-picture frames in 
the first embodiment, whereas they relate to B-picture frames in the third 
embodiment. Accordingly, the third embodiment is different from the first 
embodiment only in the operation of picture quality determination unit 1 121 in 
picture quality control unit 112. 

10 Therefore, the operation of picture quality determination unit 1121 

according to the third embodiment is explained with reference to Fig. 21. 

The calculation procedure for number of high picture quality frames 
HQ_num and high picture quality frame number HQ_frame_num[HQ_num] is 
explained with reference to Fig. 21, when the number of frames included in 

15 GOP is set to N and the frame number in picture quality determination process 
is set to i. The initial value of i is 1 and the initial value of HQ_num is 0, 

First, in step S801. picture quality determination unit 1121 determines 
whether or not the coding type of the i-th frame that is currently to be analyzed 
is B-picture. When the coding type is B-picture, the process is advanced to 

20 step S802, and when not B-picture, the process is advanced to step S804. 
In step S802, picture quality determination unit 1121 determines 
whether or not the coding type of the successive (i+1)-th frame is B-picture. 
When the coding type is B-picture, the process is advanced to step S803, and 
when not B-picture, the process is advanced to step S804. 

25 In step S803, picture quality determination unit 1 121 stores the current 

i-th frame as a frame to be encoded in a high picture quality in accordance 
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with equation (49) and updates the number of high picture quality frames in 
accordance with equation (50). 

HQ_frame_num[HQ_num] = i (49), and 

HQ_num = HQ_num + 1 (50). 
5 In step S804. picture quality determination unit 1121 increments the 

frame number i to be processed by 1 and determines whether or not frame 
number i is smaller than N. When frame number i is smaller than N, the 
process is returned to step S801 , and when not smaller, the process is 
finished. 

10 After completion of the above processes, picture quality determination 

unit 1121 turns high picture quality flag HQ_flg on when HQ_num is not less 

than 1 . and turns HQ_flg off when HQ_num is less than 1 . 

With the above process, the frame that is to be referred and is encoded 

as B-picture is regarded as a high picture quality frame number. In other 
15 words, the reference frame can be encoded in the high picture quality. Figure 

22 shows an example of B-picture encoded in the high picture quality 

according to such a process. 

Code amount control unit 111 includes frame code amount assignment 

unit 1111 and quantizing parameter update unit 1 112, similarly to the first 
20 embodiment. Explanations are given of the operations of frame code amount 

assignment unit 1111 and quantizing parameter update unit 1 1 12 in code 

amount control unit 111 according to the third embodiment. 

Frame code amount assignment unit 1111 performs frame code 

amount assignment by using the picture quality control information supplied 
25 from picture quality control unit 112, i.e., the high picture quality flag, number 

of remaining high picture quality frames R_HQ_num, and remaining high 
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picture quality frame number R_HQ_frame_num. 

Target code amounts Ti, Tp. Tb are regarded as target code amounts 
for respective picture types, R is regarded as a code amount assigned to 
frames that have not been encoded in GOP, Np and Nb are regarded as 
5 numbers of P-pictures and B-pictures that have not been encoded in GOP, Xi, 
Xp, Xp are regarded as screen complexity of each picture that is finally 
encoded, and Kp and Kb are parameters in consideration of subjective picture 
quality by picture type. Figure 23 shows the operation of frame code amount 
assignment unit 1111 according to the third embodiment, 

10 First, in step S901 , frame code amount assignment unit 1111 

determines whether high picture quality flag HQ_flg supplied from picture 
quality control unit 1 12 is turned on or off. When high picture quality flag 
HQ_flg is turned on, the process is advanced to step S 902, and when off, the 
process is advanced to step S908. 

15 In step S902, frame code amount assignment unit 1111 determines the 

frame type of the frame, which is being encoded. In a case of l-picture, the 
process is advanced to step S903, in a case of P-picture, the process is 
advanced to step S904, and in the other case, i.e., in the case of B-picture, the 
process is advanced to step S905. 

20 In step S903, frame code amount assignment unit 1111 calculates the 

code amount for (-picture, which is being encoded, in accordance with 
equation (51), and finishes the frame code amount assignment. 

Ti = R/(1 + NpxXp/(KpxXi) + NbxXb/(KpxXi)) + additional_Ti 

(51). 

25 additional_Ti = residu_bit3xXi/Xgop4 (52). 
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residu_bit3 = (margin_ratioxRx(Nb - R_HQ_num)xXb)/(KbxXgop3) 

(53). 

Xgop3 = Xi + NpxXp/Kp + NbxXb/Kb (54), and 

Xgop4 = Xi + NpxXp/Kp + (Nb « R_HQ_num)xXb/Kb (55). 
5 In this case, since the more bits are assigned to this l-frame by 

additionaLTi, in comparison with the conventional art, the picture quality of 
this frame is improved. Accordingly, the effects of the frame motion prediction 
that refers to this frame are improved. 

In step S904, frame code amount assignment unit 1111 calculates a 



10 code amount for P-picture, which is being encoded, in accordance with 
equation (56), and finishes the frame code amount assignment. 

Tp = R/(Np + Nbx KpxXb/(KbxXp)) + additionaLTp (56), 
additional_Tp = residu_bit4 x Xp/(Kp x Xgop6) (57), 
residu_bit4 = (margin_ratioxRx(Nb - R_HQ_num)xXb)/(KbxXgop5) 

15 (58), 

Xgop5 = Np X Xp/Kp + Nb x Xb/Kb (59), and 

Xgop6 = Npx Xp/Kp + (Nb - R_HQ_num)x Xb/Kb (60). 



In this case, since the more bits are assigned to this P-frame by 
additional_Tp, in comparison with the conventional art, the picture quality of 
20 this frame is improved. Accordingly, the effects of the frame motion prediction 
that refers to this frame are improved. 

In step S905, frame code amount assignmient unit 1111 determines 
whether or not frame number coding_frame_num of B-picture, which is being 
encoded, is synchronous with remaining high picture quality frame number 
25 R_HQ_frame_num supplied from picture quality control unit 112. When they 
are synchronous, the process is advanced to step S906, and when not 
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synchronous, the process is advanced to step S907. 

In step S906, frame code amount assignment unit 1111 calculates a 
code amount for B-picture, which is immediately coded, in accordance with 
equation (61), and finishes the frame code amount assignment. 
5 Tb = R/(Nb + NpxKbxXp/(KpxXb)) + additionaLTb (61), and 

additionaLTb = residu_bit4xXb/(KbxXgop6) (62). 

In this case, since the more bits are assigned to this B-frame by 
additionaLTb in comparison with the conventional art, the picture quality of 
this frame is improved. Accordingly, the effects of the frame motion prediction 
1 0 that refers to this frame are improved. 

In step S907, frame code amount assignment unit 1111 calculates a 
code amount for B-picture, which is immediately coded, in accordance with 
equation (63), and finishes the frame code amount assignment. 

Tb = (1 - margin_ratio)xR/(Nb + NpxKbxXp/(KpxXb)) (63). 
15 In this case, since the code amount assigned to this B-frame is reduced 

by margin_ratio, lowering in the picture quality of this frame is considered. 
However, with control of picture quality control unit 112, the motion prediction 
can be performed while P-. B-frames that are encoded in the high picture 
quality are referred. Accordingly, even if the code amount is small to some 
20 degree, lowering in the picture quality is prevented by improvement of the 
motion prediction performance. 

In step S908, frame code amount assignment unit 1111 determines the 
picture type of the frame, which is being encoded. In a case of l-picture, the 
process is advanced to step S909, in a case of P-picture, the process is 
25 advanced to step S9 10, and in the other case, i.e., in the case of B-picture, the 
process is advanced to step 891 1 . 
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In step S909, frame code amount assignment unit 1111 calculates the 
code amount for l-picture, which is being encoded, in accordance with 
equation (64), and finishes the frame code amount assignment. 

Ti = R/(1 + NpxXp/(KpxXi) + NbxXb/(KpxXi)) (64). 
5 In step S910. frame code amount assignment unit 1111 calculates the 

code amount for P-picture, which is being encoded, in accordance with 
equation (65), and finishes the frame code amount assignment. 

Tp = R/(Np + NbxKpxXb/(KbxXp) (65). 

In step S91 1 , frame code amount assignment unit 1111 calculates the 
10 code amount for B-picture, which is being encoded, in accordance with 
equation (66), and finishes the frame code amount assignment. 

Tb = R/(Nb + Npx KbxXp/(KpxXb) (66). 

With the above code amount assignment, without assigning a residual 
code amount to B-picture frames having low priority as reference frames, 
15 frames that are referred by another frame and are encoded as I-, P-, B- 

pictures can be encoded in the high picture quality. Accordingly, the motion 
prediction can be performed for all frames by the frames that are encoded in 
the high picture quality, and moving pictures can be encoded in the high 
picture quality, 

20 Quantizing parameter update unit 1112 obtains a quantizing parameter 

according to feedback control by macroblock, based on the virtual buffer 
capacity that is set for each picture type, in order to match code amounts 
Ti,p,b that are obtained in frame code amount assignment unit 1111 and are 
assigned to respective frames with the actual occurring code amounts. The 

25 operation of quantizing parameter update unit 1112 is similar to that of the first 
embodiment, the operation of each step is explained with reference to the 
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flowchart in Fig. 1 1 , in order to correspond to a variable name for each picture 
types. 

In step S401 , quantizing parameter update unit 1112 calculates an 
occupation amount of the virtual buffer by each picture type with equation (67) 
5 prior to encoding of the j-th macroblock. 

di,p,b(i) = di,p,b(0) + BG-1) - Ti,p,bx(j-1)/MBcount (67). 
Where, di,p,b(0) is an initial occupation amount in the virtual buffer, B(j) 
is an occurring code amount from the head of the frame to the j-th macroblock, 
and MBcount is the number of macroblocks in the frame. At completion of 
10 encoding of each frame, the initial occupation amount in the virtual buffer for 
each frame di,p,b(MBcount) is used as an initial occupation amount di,p,b(0) in 
the virtual buffer for the next picture. 

In step S402, quantizing parameter update unit 1112 calculates the 
quantization step size relative to the j-th macroblock by equation (68). 
15 Qstep = Qi,p,bxdi,p,ba)x31/(10xr) (68), and 

Qi,p,b = Xi.p,b/Ti.p.b (69). 

With control by frame code amount assignment unit 1111, assignment 
code amounts Ti,p,b of frames to be encoded in a high picture quality become 
larger than those of the conventional art, and code amount Tb of B-picture that 

20 is roughly encoded becomes smaller than that of the conventional art. 

According to equation (69), quantization step size Qb of B-picture frames that 
is encoded in the high picture quality becomes small, and quantization step 
size Qb of B-picture frames that is roughly encoded becomes large. In other 
words, the B-picture frame having high priority as a reference frame is 

25 encoded in a higher picture quality than the B-picture frame having low priority 
as the reference frame. 
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Here, r is a parameter used to control the response rate of the feedback 
loop, which is called a reaction parameter, and is represented by equation 
(70). 

r = 2 X bit_rate/frame_rate (70). 
5 Incidentally, initial values di,p(0) in the virtual buffer at start of encoding 

are represented by equations (71) to (73). 
di(0) = 10xr/31 (71), 
dp(0) = Kpxdi(0) (72), and 

db(0) = Kbxdi(O) (73). 
10 In step S 403, quantizing parameter update unit 1112 detects 

quantizing parameter Q corresponding to quantization step size Qstep from a 
quantization table. When there is no corresponding quantization step size 
Qstep in the quantization table, quantizing parameter update unit 1112 outputs 
quantization parameter Q of the quantization step size value closest to 
15 quantization step size Qstep. 

Next, explanations are given of a moving picture encoding apparatus 
according to the fourth embodiment of the present invention. In the fourth 
embodiment, l-picture, P-picture, and B-picture are used as picture types used 
to encode frames. The moving picture encoding apparatus of the fourth 
20 embodiment is configured similarly to the moving picture encoding apparatus 
of the second embodiment, however, the fourth embodiment is different from 
the second embodiment in the operations of picture quality control unit 112 
and code amount control unit 111 because encoding is performed by using B- 
pictures. In this arrangement, the operation of code amount control unit 1 1 1 is 
25 similar to that of code amount control unit 111 according to the third 

embodiment. Therefore, picture quality control unit 112 according to the forth 
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embodiment is explained below. 

Picture quality control unit 112 includes picture quality determination 
unit 1 121 and picture quality control counter 1 122, similarly to the third 
embodiment. The picture quality control unit of the fourth embodiment is 
5 different from the picture quality control unit of the third embodiment in that the 
operation of picture quality determination unit 1 121 by the frame differential 
information and the frame motion information supplied from moving picture 
analysis unit 1 12 is changed. The operation of picture quality determination 
unit 1121 according to the fourth embodiment is explained with reference to 
10 Fig, 24. 

The calculation procedure for number of high picture quality frames 
HQ_num and high picture quality frame number HQ_frame_num[HQ_num] is 
explained with reference to Fig. 24, when the number of frames included in 
GOP is set to N and the frame number in picture quality determination process 

15 is set to i. The initial value of i is 1 and the initial value of HQ__num is 0. 

First, in step S1001, picture quality determination unit 1121 calculates 
inter-frame cost IFC1(i) of each frame i and the reference frame thereof by 
using frame differential information FD1(i) and frame motion information 
FMV1(i) supplied from moving picture analysis unit 1 13 in accordance with 

20 equation (40). Also, picture quality determination unit 1 121 obtains 

IFC_UPPER_LIMIT and IFC_LOWER_LIMIT in accordance with equations 
(42) to (44) on the assumption that MINJFC(i)=IFC(i) is satisfied relative to 
MIN_|FC(i) in equation (42). On that basis, picture quality determination unit 
1 121 determines whether or not all of IFC1(i) satisfy 

25 IFC_LOWER„LIMIT<IFC(i)<IFC_UPPER_LIMIT. When all of IFC1(i) are 
within the above range, the process is advanced to step SI 002 while setting 



i=1 , and when not, the process is finished. 

In step S1002, picture quality determination unit 1121 determines 
whether or not the coding type of the i-th frame that is currently to be analyzed 
is B-picture. When the coding type is B-picture, the process is advanced to 
5 step SI 003, and when not B-picture, the process is advanced to step S1005. 
In step S1003, picture quality determination unit 1121 determines 
whether or not the coding type of the successive (i+1)-th frame is B-picture. 
When the coding type is B-picture, the process is advanced to step S1004, 
and when not B-picture. the process is advanced to step S1005. 
10 In step SI 004, picture quality determination unit 1 121 stores the current 

i-th frame as high picture quality frame number HQ_frame_num[HQ_num] in 
accordance with equation (49) and updates number of high picture quality 
frames HQ_num in accordance with equation (50). 

In step SI 005, picture quality determination unit 1 121 increments the 
15 frame number i to be processed by 1 and determines whether or not frame 

number i is smaller than number N of frames. When frame number i is smaller 
than N, the process is returned to step SI 002, and when not smaller, the 
process is finished. 

After completion of the above processes, picture quality determination 
20 unit 1 121 turns high picture quality flag HQ_flg on when HQ_num is not less 
than 1 , and turns HQ_flg off when HQ_num is less than 1 . 

Since moving picture analysis unit 113 is also arranged in the present 
embodiment, B-picture frames to be encoded in the high picture quality can be 
selected with accuracy, in consideration of momentary picture fluctuations, 
25 such as a scene change and a flash, in GOP and the effects of large 

fluctuations in encoding difficulty of the scene. Accordingly, moving pictures 
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can be encoded a still higher picture quality. 

Next, the fifth embodiment of the present invention is explained. The 
moving picture encoding apparatus according to the present invention, as 
above explained, can be configured by hardware, as is clear from the above 
5 explanations, and can be also carried out by a computer program. 

Figure 25 shows an information processing system according to the fifth 
embodiment of the present invention, and shows a typical configuration of the 
information processing system in which the moving picture encoding 
apparatus according to the present invention is implemented. 

10 This information processing system includes processor 210, program 

memory (main memory) 202, and storage media 203, 204. Storage media 
203, 204 are separate storage media and different storage areas in one 
storage medium. As a storage medium, a magnetic storage medium, like a 
hard disk drive, may be used. 

15 Then, a program for making the information processing system perform 

the above-mentioned moving picture encoding process is stored in program 
memory 202, and processor 210 executes the program, whereby this 
information processing system functions as the above-mentioned moving 
picture encoding apparatus. Input moving picture data is previously stored in 

20 storage medium 203, and the bitstreams that have been encoded are stored in 
storage medium 204. 

Therefore, the scope of the present invention includes a program for 
making a computer perform the moving picture encoding process according to 
the present invention, a computer readable storage medium stored with such a 

25 program, and a program product including such a program. 

A computer program according to the present invention, for example. 
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for making a computer that executes moving picture encoding by performing a 
multi-frame motion prediction with reference to a plurality of picture frames, 
execute: 

a process of selecting at least one reference frame among a plurality of 
reference frames of the same picture type; and 

a process of encoding the selected reference frame in a higher picture 
quality than the other reference frames of the same picture type. 

Alternatively, a computer program according to the present invention, 
for example, for making a computer that executes moving picture encoding by 
performing a multi-frame motion prediction with reference to a plurality of 
picture frames, execute: 

a process of obtaining differential information and motion information 
between a reference frame and a subject frame to be encoded; 

a process of selecting at least one reference frame among a plurality of 
reference frames of the same picture type in a manner that an interval of 
selected reference frames is adaptively changed in accordance with the 
differential information and the motion information; and 

a process of encoding the selected reference frame in a higher picture 
quality than the other reference frames of the same picture type. 

Next, as the sixth embodiment, explanations are given of an 
input/output apparatus to/from which the moving picture data or the moving 
picture bitstream, for example, encoded by the moving picture encoding 
apparatus according to the first to fifth embodiments is input and output. In 
this description, as an example of the input/output apparatus to/from which the 
moving picture data (moving picture bitstream) is input and output, 
explanations are given of a receiver that receives the moving picture encoded 
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bitstream generated according to the first or second embodiment as an input. 
However, there is no limitation on the input/output apparatus according to the 
present invention, and the input/output apparatus according to the present 
invention may be a frame synchronizer to/from which the moving picture data 
5 is input and output, a record apparatus such as a video apparatus, and the 
like. 

As shown in Fig. 26, a receiver is provided with video decoder 115 and 
reference frame picture quality monitor 116. Video decoder 1 15 is similar to 
the conventional moving picture decoding apparatus which is paired with the 

10 conventional moving picture encoding apparatus shown in Fig. 1. In other 

words, video recorder 115 includes dequantization unit 104, inverse frequency 
conversion unit 105, frame memory 106, inter-frame prediction unit 107, 
motion compensation unit 108, and buffer 110, which constitute a local 
decoding apparatus in the moving picture encoding apparatus shown in Fig. 1. 

15 Further, video recorder 115 includes variable length encoding unit 1 14. 

Functions of these elements in video decoder 115 are similar to those of the 
conventional moving picture encoding apparatus shown in Fig. 1, except 
variable length decoding unit 114. 

The video bitstream input to video recorder 115 mainly includes, as 

20 constituent elements, a conversion coefficient obtained by dividing an original 
input picture into a plurality of blocks with the moving picture encoding 
apparatus and by frequency converting and quantizing a prediction error 
signal, which a predicted value is subtracted from these blocks by the inter- 
frame prediction unit or the motion compensation unit, and a code string, 

25 which a moving vector for generating the predicted value and the reference 
frame are variable length decoded. 
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Variable length decoding unit 1 14 decodes the variable length code 
supplied from buffer 110, which is stored with the received bitstream, into 
original values. The decoded values include the conversion coefficient, the 
quantization parameter, the picture type, the motion vector, the reference 
5 frame, and the like. 

Dequantization unit 104 dequantizes the conversion coefficient by using 
the quantizing parameter and generates a dequantization conversion 
coefficient. Inverse frequency conversion unit 105 applies inverse frequency 
conversion to the dequantization conversion coefficient and generates a 
10 prediction error signal. The predicted value supplied from inter-frame 

prediction unit 107 or motion compensation unit 108 is added to the prediction 
error signal by adder 132, and a decoded picture is obtained. It can be 
selected which predicted value from inter-frame prediction unit 107 or motion 
compensation unit 108, by switch 131. The video decoder repeats these 
15 processes, thereby obtaining the decoded picture. Also, the decoded picture, 
obtained like this, is stored in frame memory 106 as a reconstructed picture 
and is used to generate the predicted value by inter-frame prediction unit 107, 
motion compensation unit 108, and motion estimation unit 109. 

Next, the operation of reference frame picture quality monitor 1 14 is 
20 explained. 

Reference frame picture quality monitor 114 monitors the picture type, 
the reference frame, the quantizing parameter, the variable length code, and 
the frame memory supplied from video decoder 115, and outputs reference 
frame picture quality fluctuation information consisting of a reference frame 
25 picture quality fluctuation flag. The reference frame picture quality fluctuation 
flag indicates that the more code amount is assigned to the reference frame 
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used for the motion compensation (multi-frame motion prediction) for decoding 
of the moving picture bitstream than other frames of the same picture type or 
whether or not a moving picture coding bitstream includes a frame having a 
small quantizing parameter. 
5 The reference frame picture quality fluctuation flag is specifically 

obtained by storing the picture type, the code amount, and the average 
quantizing parameter in decoding of the frame that is a reference frame 
referred from another frame after decoding and by checking that the more 
code amount is assigned to the reference frame stored in the memory frame 

1 0 than other frames of the same picture type or is obtained by checking whether 
or not a moving picture coding bitstream includes a frame having a small 
quantizing parameter. 

Figure 27 shows processes for generating the reference frame picture 
quality fluctuation flag, where the picture type corresponding to reference 

15 frame number i (1<i<MAX_REF) stored in the frame memory is pic_type(i), the 
code amount of the whole frame is totaLbit(i), and the average quantizing 
parameter of the whole frame is average_q(i). 

First, in step SA01 , reference frame picture quality monitor 114 
determines whether the picture type of the frame, which is being decoded, is 

20 P-picture or B-picture. When the picture type is P-picture or B-picture, the 
process is advanced to step SA02, and when neither P-picture nor B-picture, 
the reference frame picture quality fluctuation flag is turned off and the 
process is finished. 

In step SA02, reference frame picture quality monitor 1 14 determines 

25 that the more code amount is assigned to the i-th frame than the j-th reference 
frame as condition 1 , or that the quantizing parameter is small as condition 2, 
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in a plurality of reference frames ref (1<ref<MAX_REF) of the same picture 
type as the frame, which is being decoded. 
Condition 1 can be described as: 

totaLbit(i)«(1 + margin_ratio)xtotaLbit(j), pic_type(i) = pic_typeG), and 

5 i^l 

Condition 2 can be described as: 

average_q(i)«(1/(1 + margin_ratio)xaverage_q(j), pic_type(i) = 
pic_typeG), and \^\. 

In step SA02, condition 1 or condition 2 is satisfied, since the more 

10 code amount is assigned to the reference frame used by the moving picture 
bitstream for the motion prediction, than the other frames of the same picture 
type, or since such a reference frame is the moving picture encoded bitstream, 
reference frame picture quality monitor 114 tui^ns the reference frame picture 
quality fluctuation flag on. When neither condition 1 nor condition 2 is 

15 satisfied, the reference frame picture quality fluctuation flag is turned off. 

Then, in step SA03, reference frame picture quality monitor 114 
inspects the code length of the variable length code supplied from buffer 110 
to calculate code amount tmp_total_bit of the whole subject frame to be 
decoded and calculates average quantizing parameter tmp_avaerage_Q of 

20 the whole frame from the quantizing parameter by MB supplied from reference 
frame picture quality monitor 1 14. Also, at this time, picture type tmp J3c_type 
supplied from reference frame picture quality monitor 114 is stored. With 
timing in that the current subject frame to be decoded is stored in frame 
memory 106 as number k after completion of decoding, reference frame 

25 picture quality monitor 114 stores tmp_total_bit, tmp_avaerage_Q, and 

tmp_pc_type into pic_type(k), total_bit(k), and average_q(k), respectively, and 
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then the process is finished. 

The calculation of the reference franne picture quality fluctuation flag, 
like this, is performed whenever each frame is decoded. 

With the above processes, when the reference frame picture quality 
fluctuation flag is turned on, it can be confirmed that the more code amount is 
assigned to the moving picture bitstream generated by the moving picture 
encoding method according to the present invention, which is the reference 
frame used for multi-frame motion prediction, than the other frames of the 
same picture type, or that the frame having the small quantizing parameter is 
included. 



49 



